Unsupervised decoding of long-term, naturalistic human neural recordings with automated video and audio annotations

Fully automated decoding of human activities and intentions from direct neural recordings is a tantalizing challenge in brain-computer interfacing. Most ongoing efforts have focused on training decoders on specific, stereotyped tasks in laboratory settings. Implementing brain-computer interfaces (BCIs) in natural settings requires adaptive strategies and scalable algorithms that require minimal supervision. Here we propose an unsupervised approach to decoding neural states from human brain recordings acquired in a naturalistic context. We demonstrate our approach on continuous long-term electrocorticographic (ECoG) data recorded over many days from the brain surface of subjects in a hospital room, with simultaneous audio and video recordings. We first discovered clusters in high-dimensional ECoG recordings and then annotated coherent clusters using speech and movement labels extracted automatically from audio and video recordings. To our knowledge, this represents the first time techniques from computer vision and speech processing have been used for natural ECoG decoding. Our results show that our unsupervised approach can discover distinct behaviors from ECoG data, including moving, speaking and resting. We verify the accuracy of our approach by comparing to manual annotations. Projecting the discovered cluster centers back onto the brain, this technique opens the door to automated functional brain mapping in natural settings

ECoG Beta Suppression and Modulation During Finger Extension and Flexion

Neural oscillations originate predominantly from interacting cortical neurons and consequently reflect aspects of cortical information processing. However, their functional role is not yet fully understood and their interpretation is debatable. Amplitude modulations (AMs) in alpha (8–12 Hz), beta (13–30 Hz), and high gamma (70–150 Hz) band in invasive electrocorticogram (ECoG) and non-invasive electroencephalogram (EEG) signals change with behavior. Alpha and beta band AMs are typically suppressed (desynchronized) during motor behavior, while high gamma AMs highly correlate with the behavior. These two phenomena are successfully used for functional brain mapping and brain-computer interface (BCI) applications. Recent research found movement-phase related AMs (MPA) also in high beta/low gamma (24–40 Hz) EEG rhythms. These MPAs were found by separating the suppressed AMs into sustained and dynamic components. Sustained AM components are those with frequencies that are lower than the motor behavior. Dynamic components those with frequencies higher than the behavior. In this paper, we study ECoG beta/low gamma band (12–30 Hz/30–42 Hz) AM during repetitive finger movements addressing the question whether or not MPAs can be found in ECoG beta band. Indeed, MPA in the 12–18 Hz and 18–24 Hz band were found. This additional information may lead to further improvements in ECoG-based prediction and reconstruction of motor behavior by combining high gamma AM and beta band MPA

Interactive Web Application for Exploring Matrices of Neural Connectivity

We present here a browser-based application for visualizing patterns of connectivity in 3D stacked data matrices with large numbers of pairwise relations. Visualizing a connectivity matrix, looking for trends and patterns, and dynamically manipulating these values is a challenge for scientists from diverse fields, including neuroscience and genomics. In particular, high-dimensional neural data include those acquired via electroencephalography (EEG), electrocorticography (ECoG), magnetoencephalography (MEG), and functional MRI. Neural connectivity data contains multivariate attributes for each edge between different brain regions, which motivated our lightweight, open source, easy-to-use visualization tool for the exploration of these connectivity matrices to highlight connections of interest. Here we present a client-side, mobile-compatible visualization tool written entirely in HTML5/JavaScript that allows in-browser manipulation of user-defined files for exploration of brain connectivity. Visualizations can highlight different aspects of the data simultaneously across different dimensions. Input files are in JSON format, and custom Python scripts have been written to parse MATLAB or Python data files into JSON-loadable format. We demonstrate the analysis of connectivity data acquired via human ECoG recordings as a domain-specific implementation of our application. We envision applications for this interactive tool in fields seeking to visualize pairwise connectivity.Comment: 4 pages, IEEE NER 201

The reorganization of proper nouns: treatment of proper noun retrieval deficits in an individual with temporal lobe epilepsy

The neural correlates of proper noun retrieval have been investigated through neuroimaging and lesion approaches. Neuroimaging studies investigating proper noun naming in neurologically healthy individuals have demonstrated the importance of the left anterior temporal lobe (ATL) to the integrity of proper noun naming (Gorno-Tempini, 2001; Grabowski, Damasio, & Tranel, 2000; Nakamura, et al., 2000; Tranel, 2009; Tsukiura, et al., 2002), while studies investigating proper noun production in individuals with left temporal lobe lesions have demonstrated a link between left ATL damage and proper noun retrieval deficits (Damasio, Grabowski, Tranel, Hichwa, & Damasio, 1996; Tranel, 2006, 2009; Tranel, Damasio, & Damasio, 1997; Tranel, Feinstein, & Manzel, 2008; Tsukiura, et al., 2002). Though patients with left temporal lobe epilepsy have mostly normal linguistic abilities, they consistently demonstrate deficits in proper noun retrieval (i.e., famous faces and places; Glosser, Salvucci, & Chiaravalloti, 2003; Griffith, et al., 2006; Seidenberg, et al., 2002; Viskontas, McAndrews, & Moscovitch, 2002)

Molecular characterization of microbiota in cerebrospinal fluid from patients with CSF shunt infections using whole genome amplification followed by shotgun sequencing

Understanding the etiology of cerebrospinal fluid (CSF) shunt infections and reinfections requires detailed characterization of associated microorganisms. Traditionally, identification of bacteria present in the CSF has relied on culture methods, but recent studies have used high throughput sequencing of 16S rRNA genes. Here we evaluated the method of shotgun DNA sequencing for its potential to provide additional genomic information. CSF samples were collected from 3 patients near the beginning and end of each of 2 infection episodes. Extracted total DNA was sequenced by: (1) whole genome amplification followed by shotgun sequencing (WGA) and (2) high-throughput sequencing of the 16S rRNA V4 region (16S). Taxonomic assignments of sequences from WGA and 16S were compared with one another and with conventional microbiological cultures. While classification of bacteria was consistent among the 3 approaches, WGA provided additional insights into sample microbiological composition, such as showing relative abundances of microbial versus human DNA, identifying samples of questionable quality, and detecting significant viral load in some samples. One sample yielded sufficient non-human reads to allow assembly of a high-qualit

Cortical Topography of Error-Related High-Frequency Potentials During Erroneous Control in a Continuous Control Brain–Computer Interface

Brain–computer interfaces (BCIs) benefit greatly from performance feedback, but current systems lack automatic, task-independent feedback. Cortical responses elicited from user error have the potential to serve as state-based feedback to BCI decoders. To gain a better understanding of local error potentials, we investigate responsive cortical power underlying error-related potentials (ErrPs) from the human cortex during a one-dimensional center-out BCI task, tracking the topography of high-gamma (70–100 Hz) band power (HBP) specific to BCI error. We measured electrocorticography (ECoG) in three human subjects during dynamic, continuous control over BCI cursor velocity. Subjects used motor imagery and rest to move the cursor toward and subsequently dwell within a target region. We then identified and labeled epochs where the BCI decoder incorrectly moved the cursor in the direction opposite of the subject’s expectations (i.e., BCI error). We found increased HBP in various cortical areas 100–500 ms following BCI error with respect to epochs of correct, intended control. Significant responses were noted in primary somatosensory, motor, premotor, and parietal areas and generally regardless of whether the subject was using motor imagery or rest to move the cursor toward the target. Parts of somatosensory, temporal, and parietal areas exclusively had increased HBP when subjects were using motor imagery. In contrast, only part of the parietal cortex near the angular gyrus exclusively had an increase in HBP during rest. This investigation is, to our knowledge, the first to explore cortical fields changes in the context of continuous control in ECoG BCI. We present topographical changes in HBP characteristic specific to the generation of error. By focusing on continuous control, instead of on discrete control for simple selection, we investigate a more naturalistic setting and provide high ecological validity for characterizing error potentials. Such potentials could be considered as design elements for co-adaptive BCIs in the future as task-independent feedback to the decoder, allowing for more robust and individualized BCIs

Dynamic Modulation of Local Population Activity by Rhythm Phase in Human Occipital Cortex During a Visual Search Task

Brain rhythms are more than just passive phenomena in visual cortex. For the first time, we show that the physiology underlying brain rhythms actively suppresses and releases cortical areas on a second-to-second basis during visual processing. Furthermore, their influence is specific at the scale of individual gyri. We quantified the interaction between broadband spectral change and brain rhythms on a second-to-second basis in electrocorticographic (ECoG) measurement of brain surface potentials in five human subjects during a visual search task. Comparison of visual search epochs with a blank screen baseline revealed changes in the raw potential, the amplitude of rhythmic activity, and in the decoupled broadband spectral amplitude. We present new methods to characterize the intensity and preferred phase of coupling between broadband power and band-limited rhythms, and to estimate the magnitude of rhythm-to-broadband modulation on a trial-by-trial basis. These tools revealed numerous coupling motifs between the phase of low-frequency (δ, θ, α, β, and γ band) rhythms and the amplitude of broadband spectral change. In the θ and β ranges, the coupling of phase to broadband change is dynamic during visual processing, decreasing in some occipital areas and increasing in others, in a gyrally specific pattern. Finally, we demonstrate that the rhythms interact with one another across frequency ranges, and across cortical sites

Functional enhancer elements drive subclass-selective expression from mouse to primate neocortex

Viral genetic tools to target specific brain cell types in humans and non-genetic model organisms will transform basic neuroscience and targeted gene therapy. Here we used comparative epigenetics to identify thousands of human neuronal subclass-specific putative enhancers to regulate viral tools, and 34% of these were conserved in mouse. We established an AAV platform to evaluate cellular specificity of functional enhancers by multiplexed fluorescent in situ hybridization (FISH) and single cell RNA sequencing. Initial testing in mouse neocortex yields a functional enhancer discovery success rate of over 30%. We identify enhancers with specificity for excitatory and inhibitory classes and subclasses including PVALB, LAMP5, and VIP/LAMP5 cells, some of which maintain specificity in vivo or ex vivo in monkey and human neocortex. Finally, functional enhancers can be proximal or distal to cellular marker genes, conserved or divergent across species, and could yield brain-wide specificity greater than the most selective marker genes